FairDistillation: Mitigating Stereotyping in Language Models

Large pre-trained language models are successfully being used in a variety of tasks, across many languages. With this ever-increasing usage, the risk harmful side effects also rises, for example by reproducing and reinforcing stereotypes. However, detecting mitigating these harms is difficult to do general becomes computationally expensive when tackling multiple languages or considering different biases. To address this, we present FairDistillation: cross-lingual method based on knowledge distillation construct smaller while controlling specific We found that our does not negatively affect downstream performance most tasks mitigates stereotyping representational harms. demonstrate FairDistillation can create fairer at considerably lower cost than alternative approaches.